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Abstract. In this paper we study the restrictions of the minimal representation in 
the analytic continuation of the scalar holomorphic discrete series from Sp(n, M) to 
GL(n, R), and from SU(n, n) to GL(n, C) respectively. We work with the realisations 
of the representation spaces as L 2 -spaces on the boundary orbits of rank one of 
the corresponding cones, and give explicit integral operators that play the role of 
the intertwining operators for the decomposition. We prove inversion formulas for 
dense subspaces and use them to prove the Plancherel theorem for the respective 
decomposition. The Plancherel measure turns out to be absolutely continuous with 
respect to the Lebesgue measure in both cases. 

1. Introduction 

The unitary representations obtained by continuation of the scalar holomorphic dis- 
crete series of a hermitian Lie group, G, were classified by Wallach in [10], and inde- 
pendently by Rossi and Vergne Q5J). The classification amounts to membership in the 
Wallach set for the linear functionals on the compact Cartan subalgebra that extend 
the family of weights parametrising the weighted Bergman spaces on the symmetric 
space G/K. 

These unitary representations can all be realised on Hilbert spaces of holomorphic 
functions on the corresponding bounded symmetric domain S> = G/K. However, in 
this model the unitary structure cannot be described in a uniform way even though 
the corresponding reproducing kernels can. In any case, the restriction to any totally 
real submanifold defines an injective mapping. Therefore it is natural to consider an 
antiholomorphic involution r : $! — > @ that lifts to an involutive automorphism (which 
we also denote by r) of the group G. Letting H = G T denote the fixed point group, and 
L — K nH, the space 2£ := H/L is a totally real submanifold. The decomposition of 
the restriction to H of the unitary representations obtained by analytic continuation, 
or more generally, the restriction of holomorphic representations to symmetric sub- 
groups, has lately been an area of intensive research. Among those who have studied 
this problem we find, for example, Davidson, Olafsson, and Zhang ([I]), van Dijk and 
Pevzner ([9]), Zhang ([H], [12], [13j), and the author ([7], [8]). However, there does not 
yet seem to be any uniform way of dealing with this problem. For regular parameter, 
the Segal-Bargmann transform provides a unitary equivalence between the restriction 
to the group H and the left regular representation of H on the space L 2 (H/L) and 
in this case the decomposition is determined explicitly by the Helgason Fourier trans- 
form for the symmetric space H/L. Otherwise, the results obtained so far depend on 
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particular features of the special cases. In [12] . Zhang decomposes the restriction to 
the diagonal subgroup of the tensor product of a minimal representation and its dual 
by finding the spectral decomposition for the Casimir operator. The same method is 
used in [8], where the author determines the restriction of the minimal representation 
for SU(n, m) to the subgroup SO(n, m). It should be noted that this approach identi- 
fies the representations occurring in the decomposition and determines the Plancherel 
measure explicitly, but it does not provide an intertwining operator. In [7], the author 
determines the restriction from SO(2, n) to 5*0(1, n) for general parameter in the Wal- 
lach set and gives an intertwining operator. This was possible thanks to an explicit 
power series expansion for the spherical functions on the group SO(l,n). 

In this paper we consider the minimal representations for the groups Sp(n, M.) and 
SU (n, n) and restrict to the automorphism groups for the cones associated with the 
respective tube domains. We use the model in [5] that realises the representations as 
L 2 -spaces on the orbits of rank one elements on the boundaries of the respective cones. 
The intertwining operators are given explicitly as integral transforms (or, rather, as 
analytic continuations of operators defined as integral transforms). The two cases we 
deal with are identical in principle. However, the proofs are rather technical when 
it comes to parameters, so we chose to avoid a uniform presentation to increase the 
readability. Instead we present two parallel cases where the solutions follow the same 
guideline. 

The paper is organised as follows. Section 2 contains preliminaries for the two cases 
separately. In section 3 we describe the constituents in the decomposition for the 
restriction from Sp(n,M), construct an intertwining operator and prove the Plancherel 
theorem. Section 4 is the analogue of section 3 for the group SU(n, n). 

Acknowledgement: The author is grateful to his advisor Professor Genkai Zhang 
for illuminating discussions on the topic of this paper. 



2.1. Type II n . Let V be the real vector space of symmetric n x n-matrices. The 
complexification, V c = V ' ®W , of V consists of all complex symmetric n x n matrices. 
Consider the bounded symmetric domain 
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consists of the n x n blocks A, B, C and D. The isotropy group of is 
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(4) *={( £ °)|A 6 I*.)}, 
and hence 

(5) 2 = G/K. 
Let 

n = {x e v\x > o}. 

Then Q is a symmetric cone in V with automorphism group GL(n, K) acting as 
In fact 

ft = GL(n,R)/0(n). 
Moreover, the boundary of Q is partitioned into n orbits under GL(n,~R) 7 

where fiW is the set of positive semidefinite matrices of rank i. Each orbit carries a 
quasi-invariant measure, transforming in the fashion 

g*Hi = | detg\ l /j,i 

under the action of GL(n,WL). 
The Cayley transform 

c(Z) = (I - Z)(I + Z)- 1 
maps @ biholomorphically onto the tube domain 

T n := {Z = U + W e V C \U G n}. 
Let r denote the conjugation with respect to V, i.e., 

r(u + iv) = u — iv. 

The set, of fixed points of r in @, V H ^ is a totally real and totally geodesic real 
submanifold of @, and the Cayley transform restricts to a diffeomorphism 

SC = n. 

In particular, is a homogeneous space 

5£ if/L, 

where if = GL(n,M.) and L = O(n). Consider now the isomorphism 

# : GL(n, R) -> M* x SX(n,R) 

given by 

^^(det^.det^)- 1 /^) 

with inverse given by 

(A, h) V \ l,n h. 

The differential of \& at the identity element gives an isomorphism of Lie algebras 

Qi(n,R) =Re«I(n,K). 
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We will denote the SL(n, M) -factor in H by H', and correspondingly we let h' denote 
*t(n,R). 

The minimal representation in the analytic continuation of the scalar holomorphic 
discrete series of G can be defined as a Hilbert space of functions, J%/2, on Tq. This 
space has the reproducing kernel 

(6) K 1/2 (z, w) := det(z - w*)~ 1/2 . 

By (an analytic continuation of) a Laplace transform, it is unitarily and G-equivariantly 
equivalent to the the Hilbert space L 2 {QS l \ iii) (cf. [5]). Another proof of this can be 
found in [2J. We will now give an even more explicit model for this representation 
space. Consider therefore the mapping 

??:R n \{0}^ft (1) , 

defined by 

Tj(x) = XX 1 . 

Here we identify M. n with the space of all n x 1 real matrices. It is straigthforward to 
check that rj is surjective and that 

rj(x) = r](y) <^> x = ±y, 

and hence we have a bijection 

s (R«\ {0})/± 1, 

where the right hand side denotes the set of orbits under the linear action of the two- 
element group generated by the endomorphism — /. Moreover, the action of GL(n,M.) 
is covered by the linear action on R n \ {0} so that we have the following commuting 
diagram. 



R n \ {0} R n \ {0} 

V 

no) x ^ gXg \ nd) 

The measure [i\ is the pushforward under rj of the Lebesgue measure on W n \ {0}. 
Hence, the minimal representation can be realised in the Hilbert space of even square- 
integrable functions on M. n \ {0}. In this picture we have the formula 

(7) f&dethifoh*) 

for the group action on functions. Obviously this Hilbert space contains L-invariant 
functions, i.e., the representation is spherical. Therefore, by [7j, there exists a direct 
integral decomposition 

L 2 ((M"\{0})/±1)= / Hxdv, 

J A 

where A is some parameter set, the H\ are canonical representation spaces for irre- 
ducible spherical unitary representations of H' , and v is some positive measure on A, 
called the Plancherel measure for the minimal representation. 
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2.2. Type I nn . Let V be the real vector space of Hermitian n x n-matrices. The 
complexification, V c = V © iV, of V consists of all complex n x n matrices. Consider 
the bounded symmetric domain 

Q) = {Z G V C \I- Z*Z > 0}. 

The group 

G = SU(n,n) 

acts on S> by 

2h (AZ + B){CZ + D)~\ 

where 

f A B 
9= [c D 

consists of the n x n blocks A, B, C and D. We have the description 

9 = G/K. 
of 9 homogeneous space, where 

K = S(U(n) x U(n) = { ( q D^ A,D E U ^ n ^ det ( A ) det ( D ) = X } ■ 
The symmetric cone 

n = {X G V\X > 0} 
in V has automorphism group GL(n, C) acting as 

X A gXg*, 

and 

ft = GL(n, C)/U{n). 
The boundary of f2 partitions into GL(n, C) orbits as 

dQ = U" =1 fi w , 

where fiW is the set of positive semidefinite matrices of rank i. Each orbit carries a 
quasi-invariant measure, //j, transforming in the fashion 

g*Hi = | det£f| 2 >j 

under the action of GL(n, C). The Cayley transform 

c(Z) = (I - Z)(I + Z)- 1 

maps ^ biholomorphically onto the tube domain 

T n ■= {Z = U + iV e V C \U G n}. 

Let r denote the conjugation with respect to V, i.e., 

r(u + iv) — u — iv. 

The set, of fixed points of r in @, V" fl ^ is a totally real submanifold of S) and the 
Cayley transform restricts to a diffeomorphism 



6 HENRIK SEPPANEN 

In particular, 35 is a homogeneous space 

where # = GL(n, C) and L = U(n). 

The minimal representation in the analytic continuation of the scalar holomorphic 
discrete series of G is a Hilbert space, J%[, of functions on Th with reproducing kernel 

Ki(z,tu) := det(z- w*)" 1 . 

Another realisation is furnished by the Hilbert space L 2 {QS l \ni) (cf. [5], [2]). 
To give an explicit realisation, we now consider the mapping 

rj : C n \ {0} -> fi {1 ) 

defined by 

77(2:) = zz*. 

Here we identify C n with the space of all n x 1 complex matrices. It is straigthforward 
to check that rj is surjective and that 

rj(z) = rj(w) z = e^w, 

for some real 6, and hence we have a bijection 

c n /U(l). 

The action of GL(n, C) is covered by the linear action on C n so that we have the 
following commuting diagram. 



C n \ {()} C™\{0} 



fid) z ^ Zg, > fid) 
The measure /xi is the pushforward under rj of the Lebesgue measure on C n \ {0}. 
Hence, the minimal representation can be realised in the Hilbert space of [/(l)-invariant 
square-integrable functions on C n . In this picture, we have the formula 

(8) f^det R h*(foh*), 

for the group action on functions, where the subscript on the determinant means the 
determinant of h as an M-linear operator on R 2n . 

3. The branching rule: type II n 

3.1. Some parabolically induced representations . In the following, we will con- 
sider some parabolically induced representations of H' = SL(n,M). 
Let do = Me, where 




where l n -\ denotes the identity matrix of size (n — 1) x (n — 1). The maximal parabolic 
subalgebra, q , determined by do has a decomposition 



q = n © m © a © n , 
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where 



/ xi ■■■ x n -x \ 
••• 



n 



m 



n n = 



\ 


M 





/ 



\X\i . . . , x n ^\ G 



\m esi(n- 1,: 



/ 

xi 







|Xx, . . . , X n _ 1 £ 



: : ■•. 
V x n _i • • • / 

Here the subspace m is defined by the property 

^f)'(°o) = dO © 1Tl , 

and 

n = {X e fj'| = a(#)X, V# G a } , 

Tio = {Xefj'l = ~a(H)X, V#Ga } 

are the generalised root spaces, where the root ct G a * is determined by 

a(e) = n. 

We let p denote the half sum of the positive roots counted with multiplicity, i.e., 

n — 1 



Po = 



-a. 



On the group level we have the corresponding decomposition 

Q = M A N , 



where 



A = 
M = 

N = { 



e s 

g/ n _! 

1 
M 



|s,g G R,eV -1 = 1 



M G - 1, 



' / 1 Xi x 2 • 

1 • 

1- 

: : " 



X„-l \ 






1 / 



\X\ , . . . , x n —\ G 



^ \ 

Consider now the representation 1 ® exp iA <S> 1 of the group 

Qo = M A N . 

The induced representation 

(9) r A := Indg(l <8> exp(iA + p ) <g> 1) 
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has a noncompact realisation in the Hilbert space L 2 (N , dn) (cf. [4]). We have 

(10) ir\(h)f(n) = e~ {iX+po) ^ aa ^ lll)) f{n^{h- l n)), 

The decomposition f)' = © m © a © no gives a corresponding decomposition 

W = W M A N , 

by which we mean that the equality holds outside a set of strictly lower dimension. 
The factorisation of a group element with respect to this decomposition is not unique, 
but the A -component is. For h G H', we let ao(h) denote this component. The 
mapping exp : a — > A is a diffeomorphic homomorphism of abelian groups. We let 
log : Aq — ► Oo denote its inverse. The representation T\ is then given by 

(11) r x (h)f(n) = e- {a+po){losao( ^ %)) /(r^(/i- 1 n)). 
For arbitrary h G H',n G No, a factorisation of hn can be given by 

a b \ ( 1 

C d J I X I n _l 



c+dx 
a+bx 



n \ / a+bx n 
U » ' |a+6x| 



In-x J V \a + bx\ 1 / n ~\d- (gg)6) 



x la + bxl °u , H 1 ^ 

In view of this, and identifying L 2 (N , dn) with L 2 (IR n_1 , dx), we obtain the following 
explicit formula for the induced representation: 



c + dx 
a + bx 



(r x (h)f)(x) = \ax + b\- (lX+n ^f 

where h — ( ^ 
y c d 

3.2. An intertwining operator. Let m x be the one-dimensional representation 

m x (c)z := \c\ iX+po z 
of M*. Recalling the isomorphism 

* : R) -> R* x SX(n, R) 

from the previous section, we can now form the representation 

(12) TT\ '.= VTL\ © T\ 

of GL(n, R). We let 7Ya denote the associated representation space. 

We now consider an operator, T, mapping a C^°((M ?1 \ {0})/ ± l)-function, /, to a 
function Tf : C x R™ -1 — > R which is meromorphic in the first variable. The function 

(13) Tf(z, V ):= [ f(x)\(x,(l, V ))\-^ +n ^dx 
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is well defined as a function of z and rj when Imz — n/2 > —1. For such z, a change of 
variables, followed by an integration by parts, yields 

f(x)\{x,(l,r,))\-^+^dx 
1 



(1 + \n\ 2 ) iz + n / 2 (-(iz + n/2) + 1) 
* ° i 

l yi <o 9yi 



(1 + \n\ 2 ) iz+n / 2 (-(iz + n/2) + 1) 



> yi >o 9yi 

where g is some orthogonal transformation such that 

(1,77) = 0(1(1, i7)| ei ). 
By repeated integration by parts we get the identity 
(14) Tf(z, V ) 

1 



(1 + \r]\ 2 ) iz+n l 2 T{ k j=1 (-(iz + n/2) + j) 
111 

1 m ii ^Wl 



+(-l) fc 



1 + \n\ 2 ) iz+n l 2 Ii k j=l (-(iz + n/2) + j) 

Therefore, Tf(z,n) can be continued to a meromorphic function with poles at z = 
i{n/2 — j), for j = 1,2,.... In particular, Tf(X,n) is a well defined real analytic 
function for real A (except possibly for A = 0). We write T\f(n) for Tf(X,n). 

Proposition 1. For f G C °°((]R" \ {0})/ ± 1) and A G 1, the functions T x f is m 



Proof. For A G M, choose the natural number k > n/2 in f TTil ). The functions 

dy\ 

constitute a uniformly bounded family in the supremum-norm. Hence, we have an 



(15) !TkU°h), heO(n) 



estimate 



2\-(i\+n/2) 



(16) |T A /fo)|<C(A)(l + |i7 

and this proves the claim. □ 

In what follows, we will state and prove properties for the functions Txf for arbitrary 
real A although the proofs will use the defining integral (fl"3l) which makes sense only 
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when Imz > n/2 — 1. The idea is then that both sides in the stated equalities are 
meromorphic functions, so by the uniqueness theorem for meromorphic functions it 
suffices to perform the calculations when the defining integral makes sense. All integral 
equalities should therefore be thought of as analytic continuations of the corresponding 
equalities when the integrals are convergent. 

Proposition 2. The operator 

T x :C™(R n /{±l})^H x , 

given by 

(17) T x f(rj)= [ f(x)\(x,(l,r,))\-^ +n ^dx 

is H-equivariant. 

Proof. Take g G H and write g = (h, where ( is a diagonal matrix and h has determi- 

a b 
c d 

Ugf)(v) 

/(^)|((x 1 ,x / ),(l,r 7 ))|-^ + "/ 2 )det^x 



nant 1. Moreover we write h 1 = ( \ ) . Then 



|Cr A+n/2 / f(x)\a + br]\^ iX+n ^ 



x|((x 1 ,x , ),(l,(c + C /r / )(a + ^)- 1 ))|-( lA+ "/ 2 ) C /x 
n\(g)T x f(v)- 



□ 



If / is L-invariant, then T\f is an L-invariant function in the representation space 
TC\. By the Cartan-Helgason theorem ([3j), the subspace of L-invariants is at most one 
dimensional. In fact, it is spanned by the function r\ \— > (1 + |r7| 2 )~(* A+n / 2 )/ 2 . Thus, we 
can define a function / by 

(18) T x f(r ] ) = f(\)(l + \r ] \ 2 )-^ n ^ 2 . 

The plan is now to prove an inversion formula and a Plancherel theorem. The 
following lemma will be very useful in the sequel. 

Lemma 3. Let f 6 C^ C, (R™ \ {0}) L . Then the function f can be written in the form 
~ _ /2 r(-(2»A + (n-2))/2) r^f(reAMX) 

/(A)_27r r(i/2)r((-2a + n)/4)^ (r r H rei >>W> 

where j$ is the Mellin transform. 
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Proof. We start by observing that, since / has compact support outside the origin, 
the Mellin transform above admits an entire extension by the Paley- Wiener theorem. 
It thus suffices to prove the statement for A G i(n/2 — 1, oo) by the uniqueness of an 
analytic continuation. 

By the L-invariance of / and of the Lebesgue measure, we have 

(19) f f(x)\(x,(l, V ))\-^ +n ^dx. 

= f f(x) [ \{gx,(l,r,))\-^ +n ^dgdx. 

JH" J SO(n) 

Consider now the function 

(20) R(x,y):= [ \(gx,y)\-^ +n ^dg, x,y eW\ 

J SO(ri) 

It is S , 0(n)-invariant in each variable separately, and it is homogeneous of degree 
— (iX + n/2). Hence, 

(21) R{x,y) = \x\- {lX+n/2) \y\- {iX+n/2) f {(ge^e^-^^dg. 

JSO(n) 

The integral on the right hand side can be expressed as an integral over the sphere 
S' Tl ~ 1 . Indeed, the fibration 

(22) p:SO(n)^S n - l ,p(g) = ge 1 

defines a measure a on S 1 ™ -1 as the pushforward of the normalised Haar measure on 
SO(n), i.e., a is defined as an SO{n) -invariant linear functional on C(S n ~ r ) by the 
equation 

(23) / /(OMO := / f(p(9))dg, f G CiS"- 1 ). 

JS™- 1 JSO(n) 

By choosing / as a constant function in the above equality, we see that a is the 
normalised surface measure on 6' n_1 . Applying (1231 ) to the equality ([211 . we get 

(24) R(x,y) = | x |-(^+n/2)| y |-(iA+n/2) f |^ | -(iX+n/2) _ 

Js™- 1 

The last integrand depends only on one variable, and hence we can apply the "Functions 
of fewer variables'-theorem (cf. [6]) and replace the integral by an integral over the 
unit interval on the real line. This yields 

f \Ci\- {iX+n/2) da(C) 
Js™- 1 



2V{n/2) 
r(l/2)r((n-l)/2) 
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By performing the change of variables s = 1 — t 2 , we obtain 

2r(n/2) f\ 1 _ f ^ t _ (iX+n/2)dt 



r(l/2)r((n - l)/2) 7 

r(n/2) n-1 i ■ , 2iA+Q-2) 1 

' / ~\ / s 2 (1 - s) 4 L ds 



r(l/2)r((n - l)/2) 7 

r(n/2) ^Z^" 1 2iA + (n-2) 



r(l/2)r((n-l)/2)" V 2 
r(n/2)r(-(2a+ (w-2))/2) 
r(l/2)r((-2iA + n)/4) 



and hence 
(25) 

Inserting (1241) (with y = (1, 77)) and (l25l) into (TT9l) gives 



|-(u + n/ 2 ) , rn = r(n/2)r(-(2tA + (n-2))/2) 
M lU r(l/2)r((-2iA + n)/4) ' 



n , 1 ,2x- ( q + n/2)/2 E (n/2)r(-(2»A + (n - 2))/2) 
1 r(l/2)r((-2iA + n)/4) 

x / /(x)|xr (lA+n/2) rfx. 



Finally, we use polar coordinates to compute the integral on the right hand side. Then 

f(x)\x\-^^dx = /°°r"/ 2 /(r ei )r- iA -, 
1 \ n l A ) Jo r 

and hence the lemma is proved. □ 
Theorem 4 (Inversion formula). If f e C °°(M n \ {0}) L , then 

fire,) = T -^± [ f(\y^ r ((-2iA + n)/4) 
/(r6lj 2W> r((-(2.A + (n-2))/2) dA - 

Proof. By the previous lemma, we can write 

(26) J({r » r"/ 2 /(r ei ))(A) := /(A)6(A). 

By the assumptions on /, the inverse Mellin transform is defined for the left hand side 
since it is in L 1 , and the inversion formula for the Mellin transform yields 

(27) W 2 /(r ei ) = / f(\)b(\y x - l d\, 

Jr 

i.e., 

M u 2W 2 r((-(2iA + (n-2))/2) 

□ 
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Remark. Note that this is a somewhat peculiar looking "Inversion formula". It does 
not express the function / as a weighted superposition of some canonical functions 
with respect to the Plancherel measure for the given representation. This will become 
clear by the next theorem. The reason that we prove it is rather because it serves as 
a means for proving the Plancherel theorem. 

Theorem 5 (Plancherel theorem). For all f G C^(R n \ {0}) L we have 
/ \f(x)\ 2 dx = / \f(X)\ 2 

JR n JR 

Proof. We introduce some temporary notation and write the inversion formula in the 
simplified form 

/(rei) = I f{\y x ~ n ' 2 <P(X)d\. 

JR 

By the inversion formula we then have 

/ \f(x)\ 2 dx = [ f(x) [ hX)\A~ lX - n/2 W)dXdx 

JR n JR n JR 

= /7(a)/ f{x)\xr x - n ' 2 dxW)d\- 

JR JR n 

By the proof of Lemma[3l the inner integral can be seen to be equal to /(A)0(A), and 
hence 

/7(A)/ f(x)\xr x - n / 2 dxmd\= [ \f(\)\ 2 \m\ 2 d\, 

JR JR" JR 

and this concludes the proof. □ 
Theorem 6. The operator T extends to a unitary H -intertwining operator 

(28) U : L 2 ((R n \ {0})/ ± 1) - / W A t^(A), 

JR 

where fj, is the measure determined by the identity 

[ f(X)dfi(X) := / /(A) 
JR JR 

Proof. By Prop. [2] and Thm. [5l there exists a unique if -intertwining extension U : 
L 2 ((R n \ {0})/± 1) -> j R Hxdfi{\) of T. The only thing that remains to prove is the 
surjectivity of U. 

This follows immediately from the proof of Theorem 9 in [7]. Indeed, by the H- 
equivariance of the operator U the action of the commutative Banach algebra L 1 (if) # 
of left and right L-invariant /^-functions on H is intertwined. On each subspace 
7Y^, a function / G L 1 (ii)# acts as a scalar operator, /(A). If we let V\ denote the 
canonical L-invariant vector associated with the spherical representation on 7ix, and 
uj G L 2 ((M. n \ {0})/ ± 1) denote an L-invariant vector in the minimal Lf-type, then the 
positive functional $ on L l (H)# given by $(/) = (ir(f)uj,uj) can be written as the 
integral 

(29) $(/) = / MfWW, 

JR 



r(l/2) r((-2zA + n)/4) 
2W 2 T((-(2zA + (n-2))/2) 



dX. 



r(l/2) T((-2iX + n)/4) 
2W 2 r((-(2zA+(n-2))/2) 
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where <p\ is the multiplicative functional / i— > (f(\)v\,V\)\. The surjectivity now fol- 
lows from the proof of Theorem 9 in [7] by uniqueness of such an integral decomposition 
of$. ' □ 

4. The branching rule: type I nn 
We consider the diffeomorphism 

* : GL(n,C) -> C* x SX(n,C) 

given by 

^(det^.detfo)- 1 ^), 
where we have chosen the branch of the nth-root multifunction determined by the root 
of unity with the least argument (i.e. in polar coordinates (re i6, ) 1//n := r l / n e ie / n y The 
mapping \I/ has inverse 

: (\h) ^ \ l/n h. 

In this case, however, \1> is not a group homomorphism since the chosen branch of the 
multifunction is not multiplicative. Instead \I/ is multiplicative up to scalar multiples of 
modulus one. We shall see later that we can still use this diffeomorphism to construct 
representations of GL(n, C) from representations of SL(n, C) and C* respectively. 

4.1. Some parabolically induced representations of SL(n,C). Let a = Me, 
where 



n-1 

-I n -! 

Consider the maximal parabolic subalgebra, q , determined by do, with decomposition 

q = no © Wo © d © n , 



where 



f z l ■■■ z n _ x \ 
■•• 



m 



n 



\ 


M 





/ 



\Zl, ■ ■ ■ , z n-l £ C 



\M e sl(n — 1, C) > , 



/ ••• \ 

zi ••• 



: : ■•• 

\ Zn-l • ■ ■ / 

Here the subspace m is defined by the property 

Z v {oq) = a ©m , 

and 

{X G fj'| [#,X] = a(H)X, 



z n -i e C 



no 
n? 



V# g a } 



{X G fj'| [if, X] = -a(H)X, VH G a } 
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are the generalised root spaces, where the root a G a * is determined by 

a(e) = n. 

We let po denote the half sum of the positive roots counted with multiplicity, i.e., 

p = (n- l)a. 

On the group level we have the corresponding decomposition 

Q = M A N , 
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where 



A 
M 

iV 



e s 

ql n -i 

1 
M 



\s,qe R,e s g n_1 = 1 



M eGL(n-l,C) \, 



( ( 1 z l z 2 

1 

1 

; ; o 





\ 

















1 


/ 



\Zl, ■ ■ ■ , Zn-1 ^ C 



t \ o o o 

Consider now the representation 1 <S> exp iX <g) 1 of the group 

Qo = M A N . 

We realise the induced representation 

(30) r A := Ind^(l ® exp(iA + p ) <8> 1) 

in the Hilbert space L 2 (N Q , dn). 

The decomposition on the group level 

H' = W M A N , 

gives that for h G H',n G iVo, hn can be factorised as 



a b 
c d 



1 

z I n -i 



1 

-dz j 

J-n-l 



X 



a+bz 

\a + bz 




a+bz 







|a+fez| 

\a + bx\ 1 ' n - 1 (d- (gg)6) 

6' 







1 



Hence, by identifying L 2 (N , dn) with L 2 (C n_1 , dm(z)), where dm(z) is the Lebesgue 
measure on C n , we obtain the following formula for the action of H' on functions in 
the representation space: 

r x (h)f(z) = \az + b\'^f r ' '"" ^ 



a + bz J 



where /i 



a 6 
c d 
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4.2. An intertwining operator. Recalling the diffeomorphism 

* : GL(n,C) -> C* x SX(n,C) 

from the previous section, we can now form the representation 

m A ® IndQ (l <g> exp(zA + p ) ® 1), 

where m>(c) = |c| lA+p , of C* x SL(n,C). This will in fact give a representation of 
GL(n,C). Indeed, suppose that we have <?i,g 2 € GL(n,C). We can write 

= A/ hi, 

and 

#2 = A 2 7 Al 2 , 
with Ai, A 2 e C* and fri, /i 2 e SX(n, C). Then 

^2 = (AiA 2 ) 1 / n e(Ai,A 2 )/i 1 /i 2 , 

where |£(Ai,A 2 )| = 1 and hence the mapping 

(31) 7T\ '. g i — ► Tfi\ ® Ind^(l <g> exp(zA + p ) ® 1) ° 

defines a unitary representation of GL(n,C). We let 7-^ denote the corresponding 
representation space. 

For / e C °°(C n ), we define the function T/:Cx C n_1 -> C by 

Tf(\,v):= I f(z)\(z,(l, V )\-^dm(z), 

where the right hand side is to be interpreted using analytic continuation in the vari- 
able A as in the previous section (eq. (fl3l) and the following discussion). We have the 
following analog of Prop. [TJ 

Proposition 7. For f G C£°((C n \ {0})/ ± 1) and HI, the functions T x f is in 
L 2 {C n - r ). 

The proof is the same as that of Prop. Q3 

All the following integral equalities where the variable A occurs are to be thought of 
as analytic continuations of the corresponding equalities involving convergent integrals. 
We write T\f for the function 77 1— > T/(A, rj). 

Proposition 8. The operator 

(32) T\ : C °°((C" \ {0})/C/(l)) -> H x 
is H-equivariant. 
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Proof. Take g G H and write g = (h, where ( is a diagonal matrix and h has determi- 

a b 
c d 



nant 1. Moreover we write h 1 = ( ^ ) . Then 



Tx(9f)(v) = / f(g*z)\((zi,z'),(l, V ))\-^\detg*\ 2 dm(z) 
= [ f{x)\{{z^z'),g-\l^))\-^dm{z) 
= \C\ lX+n [ f{x)\{a + b n )z 1 + (z',c + dri)\-( iX+n ^dm(z) 

= \C\ lX+n [ f(x)\a + bn\^ iX+n) 

x \{( Zl , z'), (1, (c + d V ){a + bq)- 1 )) \- {lX+nl2) dm(z) 
= 7r x (g)T x (f)( v ). 



□ 



For an L-invariant function /, the function r] i— > Tf(X J 7]) is L-invariant by the 
above proposition. The Cartan-Helgason theorem ([3j) therefore allows us to define 
the function / by 

T(A,77) = /(A)(l + | 77 |2)-(iA+rx)/2_ 

Lemma 9. Let f € C^°(C n \ {0}) L . TTien i/ie function f can be written in the form 



r 



i\+(n-2) \ 

2 y 



(33) /(A) = 47T- ? . x V nN ^^(r - rV(reO)(A) 



r (^- a+( 2 w ~ 2) 4 // - i 

where <M is the Mellin transform. 

Proof. The proof is almost identical to that of Lemma [3l We assume that A is purely 
imaginary with big enough imaginary part. Then 

(34) [ f(z)\(z,(l,r)))\-^dm(z) 



f(z) / \(gz,(l, V ))\-^dgdm(z). 

C" JSU(n) 

The inner integral can be written as 
(35) f f(z)\(z,(l,r,))\-^ +n Um(z) 

f(z) [ \(gz,(l,r])}\-^dg 

C" JSU(n) 
| ;g |-(iA+«)( 1+ | 77 |2)-(iA+n)/2 f ^- { iX+n) da ^y 

Js 2 ™- 1 
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The integrand on the right hand side depends only on one variable, and hence we can 
apply |6J, Prop. 1.4.4. This yields 

(36) f |Ci|- (4A+n W(C) 

Js 2 "- 1 

= ^Zl [ (i _ \ z \ 2 ) n - 2 \ z \^ x+ ^dm(z), 
k Ju 

where U is the unit disc in C. The last integral can be written as 

(37) — - / (1 - \z\ 2 ) n - 2 \z\~ {lX+n) dm(z) 

K Ju 



2vr(n - l)(3{n - 1, -(iA + (n - 2))/2) 

-p i i\+(n—2) 

2tt- 



tA+(n-2) 
2 

Using (l35l) . (|36l) . and f|37l) . the identity (1341) can be rewritten in the form 

-p / iX+(n— 2) " 
2\-(iA+n)/2_ V 2 



C™ 



47T n (l + |r/| 2 ) 

r(-^+p^ + n-i 

x / /(z)|z|^ A+ri W(z). 



Using polar coordinates, the integral on the right is given by 

2rr n r°° dr 
f(z)\z\-^dm(z) = — / r"/(r ei )r- lA -, 

r(n) y r 

and this proves the statement. □ 

Since / has compact support outside the origin, the right hand side admits an 
extension to an entire function by the Paley- Wiener theorem. 

Theorem 10 (Inversion formula). If f e C °°(C n \ {0}) L , then 

where (•)& denotes the Pochhammer symbol defined as 

(t)o = 1, 

(t) k = t{t + l)---{t + k- 1), k e N+. 

Proof. The identity 

. , T(x + k) 
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shows that the statement of Lemma [9] can be written in the form 
(38) ^(r^r"/(re 1 )) = i ^(- ZA+ 2 n ~ 2 ^) /(A) 

The inversion formula for the Mellin transform then yields the identity 
/(„,) = -L f f W r a -"dX. 



4vr" 



n-l 



□ 



Theorem 11 (Plancherel theorem). For all f G Co°(C n \ {0}) L we have 



C" 



\f{z)\'dm{z) = (i) / |/(A)p 



zA + n - 2 

i / /-(Air i - 

4ir n 

Proof. For simplicity, we write the inversion formula in the form 

/(r ei ) = / f(\)r iX ~ n <fi(\)d\. 
Jr 

By the inversion formula we then have 



n-l 



2 



|/(«)|'dm(z) = / f{z) / /(A)| 2 |-^>(A)dm(z) 



/(A) / fiz^zr^dmiz^dX. 

By the proof of Lemma[9l the inner integral can be seen to be equal to /(A)0(A), and 
hence 



/(A) / f(z)\zr x - n dm(z)4>(X)dX= / \f(X)\ 2 \4>(X)\ 2 dX, 

and this concludes the proof. □ 

By the same argument that we used to prove Theorem EJ we have the following 
branching law. 

Theorem 12. The operator T extends to a unitary H -intertwining operator 
(39) U : L 2 ((C n \ {0})/U(l)) - / H x d^(X), 



where \i is the measure determined by the identity 



f{X)dn{X) := ( — ) / /(A) 



«A + 72 - 2 



2 



dA. 



Remark. The ideas in this paper could probably be extended to the case of the type 
1 1 I n bounded symmetric domain consisting of complex antisymmetric n x n matrices 
by realising the corresponding minimal representation as the Hilbert space 
L 2 ((HP \ {0}) / Sp(l)) and proceeding in an analogous way. 
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